Papers with Connectionist Temporal Classification
CTC-based Compression for Direct Speech Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing studies have shown that a dynamic phone-informed compression of the input audio is beneficial for speech translation (ST). |
| Approach: | They propose a method which performs a phone-informed compression of the input audio in direct ST models by exploiting the Connectionist Temporal Classification (CTC) they demonstrate that their method brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German) |
| Outcome: | The proposed method brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German) it reduces memory footprint by more than 10%, and is faster than previous approaches. |
CTC Alignments Improve Autoregressive Translation (2023.eacl-main)
Copied to clipboard
Brian Yan, Siddharth Dalmia, Yosuke Higuchi, Graham Neubig, Florian Metze, Alan W Black, Shinji Watanabe
| Challenge: | Connectionist Temporal Classification (CTC) is widely used for automatic speech recognition (ASR) but lags behind attentional decoder approaches in terms of translation quality. |
| Approach: | They propose to use a CTC/attention framework to validate this hypothesis by modifying the Hybrid CTC-Attention model proposed for automatic speech recognition to support text-to-text translation (MT) and speech-totext translation. |
| Outcome: | The proposed model outperforms pure-attention baselines across six translation tasks. |
A Study of Syntactic Multi-Modality in Non-Autoregressive Machine Translation (2022.naacl-main)
Copied to clipboard
| Challenge: | Non-autoregressive translation models suffer from the multi-modality problem when a source sentence corresponds to multiple correct translations. |
| Approach: | They propose to decompose the syntactic multi-modality problem into short- and long-range models and evaluate them on synthesized and real datasets. |
| Outcome: | The proposed loss functions can handle short- and long-range syntactic multi-modalities better than existing models. |
Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | Developing techniques to support end-to-end speech translation is non-trivial because of the speech-text modality gap. |
| Approach: | They propose a coarse labeling approach that merges vocabulary labels via simple heuristic rules . they propose to use 256-bit truncation, division or modulo operations to regularize the encoder . |
| Outcome: | The proposed method can increase training efficiency while delivering better performance. |
Improvement in Sign Language Translation Using Text CTC Alignment (2025.coling-main)
Copied to clipboard
| Challenge: | Current sign language translation (SLT) approaches rely on gloss-based supervision with Connectionist Temporal Classification (CTC) limiting their ability to handle non-monotonic alignments between sign language video and spoken text. |
| Approach: | They propose a method that integrates CTC/Attention with the attention mechanism during decoding and integrates it with the sign language video and spoken text. |
| Outcome: | The proposed method outperforms the pure-attention baseline and achieves comparable results to state-of-the-art methods. |
Comparison of Conventional Hybrid and CTC/Attention Decoders for Continuous Visual Speech Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances have been achieved in Visual Speech Recognition (VSR) despite the lack of data, there is no clear comparison between different types of decoders for certain languages and tasks. |
| Approach: | They focused on how the conventional DNN-HMM decoder behaves depending on the amount of data used for their estimation. |
| Outcome: | The proposed model improves the CTC/Attention model in data-scarcity scenarios while requiring less training time and fewer parameters. |
Massive End-to-end Speech Recognition Models with Time Reduction (2024.naacl-long)
Copied to clipboard
Weiran Wang, Rohit Prabhavalkar, Haozhe Shan, Zhong Meng, Dongseong Hwang, Qiujia Li, Khe Chai Sim, Bo Li, James Qin, Xingyu Cai, Adam Stooke, Chengjian Zheng, Yanzhang He, Tara Sainath, Pedro Moreno Mengibar
| Challenge: | Using the neural architecture of Google’s universal speech model, we reduce the frame rate and speed up training and inference. |
| Approach: | They propose to use the neural architecture of Google’s universal speech model with additional funnel pooling layers to significantly reduce the frame rate and speed up training and inference. |
| Outcome: | The proposed methods work with both connectionist temporal classification (CTC) and RNN-Transducer (RNN-T) and over two domains. |
OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification (2024.acl-long)
Copied to clipboard
| Challenge: | Autoregressive models can be slower during inference and have potential risks of hallucination. |
| Approach: | They propose an encoder-only speech foundation model based on Connectionist Temporal Classification. |
| Outcome: | The proposed model improves on 180k hours of public audio data for multilingual speech recognition, speech translation, and language identification. |
LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained speech encoders and large language models can perform suboptimal performance on a range of spoken language processing tasks. |
| Approach: | They propose to combine large-scale pre-trained speech encoders and large-language models for better performance on automatic speech recognition tasks. |
| Outcome: | The proposed model can get an average of 49% WER reduction over the baseline model on 8 MLS testsets. |